{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WfGpInivS0fG"
   },
   "source": [
    "<h2 align=\"center\">点击下列图标在线运行HanLP</h2>\n",
    "<div align=\"center\">\n",
    "\t<a href=\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_stl.ipynb\" target=\"_blank\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\t<a href=\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fner_stl.ipynb\" target=\"_blank\"><img src=\"https://mybinder.org/badge_logo.svg\" alt=\"Open In Binder\"/></a>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IYwV-UkNNzFp"
   },
   "source": [
    "## 安装"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Uf_u7ddMhUt",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pp-1KqEOOJ4t",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install hanlp -U"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0tmKBu7sNAXX",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 加载模型\n",
    "HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EmZDmLn9aGxG",
    "outputId": "0d55f7a1-3a4c-4170-e60f-da7473208e3f",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'MSRA_NER_BERT_BASE_ZH': 'https://file.hankcs.com/hanlp/ner/ner_bert_base_msra_20211227_114712.zip',\n",
       " 'MSRA_NER_ALBERT_BASE_ZH': 'https://file.hankcs.com/hanlp/ner/msra_ner_albert_base_20211228_173323.zip',\n",
       " 'MSRA_NER_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/ner/msra_ner_electra_small_20220215_205503.zip',\n",
       " 'CONLL03_NER_BERT_BASE_CASED_EN': 'https://file.hankcs.com/hanlp/ner/ner_conll03_bert_base_cased_en_20211227_121443.zip'}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import hanlp\n",
    "hanlp.pretrained.ner.ALL # 语种见名称最后一个字段或相应语料库"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VDT-qmLyvDST"
   },
   "source": [
    "调用`hanlp.load`进行加载，模型会自动下载到本地缓存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "Tzu5Qi-xvDST",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading https://file.hankcs.com/hanlp/ner/msra_ner_albert_base_20211228_173323.zip to C:\\Users\\COLORFUL\\AppData\\Roaming\\hanlp\\ner/msra_ner_albert_base_20211228_173323.zip\n",
      "100%  37.5 MiB 804.6 KiB/s ETA:  0 s [=========================================]\n",
      "Decompressing C:\\Users\\COLORFUL\\AppData\\Roaming\\hanlp\\ner/msra_ner_albert_base_20211228_173323.zip to C:\\Users\\COLORFUL\\AppData\\Roaming\\hanlp\\ner\n",
      "Failed to load https://file.hankcs.com/hanlp/ner/msra_ner_albert_base_20211228_173323.zip\n",
      "Please upgrade HanLP with:\n",
      "\n",
      "\tpip install --upgrade hanlp\n",
      "\n",
      "If the problem persists, please submit an issue to https://github.com/hankcs/HanLP/issues/new?labels=bug&template=bug_report.md\n",
      "When reporting an issue, make sure to paste the FULL ERROR LOG below.\n",
      "================================ERROR LOG BEGINS================================\n",
      "OS: Windows-10-10.0.26100-SP0\n",
      "Python: 3.9.10\n",
      "PyTorch: 2.1.2+cpu\n",
      "TensorFlow: 2.13.1\n",
      "HanLP: 2.1.1\n"
     ]
    },
    {
     "ename": "OSError",
     "evalue": "Can't load tokenizer for 'uer/albert-base-chinese-cluecorpussmall'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'uer/albert-base-chinese-cluecorpussmall' is the correct path to a directory containing all relevant files for a BertTokenizer tokenizer.\n=================================ERROR LOG ENDS=================================",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[5], line 3\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;66;03m# ner = hanlp.load(hanlp.pretrained.ner.MSRA_NER_ELECTRA_SMALL_ZH) # MSRA_NER_ELECTRA_SMALL_ZH\u001b[39;00m\n\u001b[0;32m      2\u001b[0m \u001b[38;5;66;03m# ner = hanlp.load(hanlp.pretrained.ner.MSRA_NER_BERT_BASE_ZH)\u001b[39;00m\n\u001b[1;32m----> 3\u001b[0m ner \u001b[38;5;241m=\u001b[39m \u001b[43mhanlp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mload\u001b[49m\u001b[43m(\u001b[49m\u001b[43mhanlp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpretrained\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mner\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mMSRA_NER_ALBERT_BASE_ZH\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\hanlp\\__init__.py:43\u001b[0m, in \u001b[0;36mload\u001b[1;34m(save_dir, verbose, **kwargs)\u001b[0m\n\u001b[0;32m     41\u001b[0m     \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01mhanlp_common\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mconstant\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m HANLP_VERBOSE\n\u001b[0;32m     42\u001b[0m     verbose \u001b[38;5;241m=\u001b[39m HANLP_VERBOSE\n\u001b[1;32m---> 43\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m load_from_meta_file(save_dir, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmeta.json\u001b[39m\u001b[38;5;124m'\u001b[39m, verbose\u001b[38;5;241m=\u001b[39mverbose, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\hanlp\\utils\\component_util.py:188\u001b[0m, in \u001b[0;36mload_from_meta_file\u001b[1;34m(save_dir, meta_filename, transform_only, verbose, **kwargs)\u001b[0m\n\u001b[0;32m    186\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m:\n\u001b[0;32m    187\u001b[0m     \u001b[38;5;28;01mpass\u001b[39;00m\n\u001b[1;32m--> 188\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m e \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\hanlp\\utils\\component_util.py:106\u001b[0m, in \u001b[0;36mload_from_meta_file\u001b[1;34m(save_dir, meta_filename, transform_only, verbose, **kwargs)\u001b[0m\n\u001b[0;32m    104\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m    105\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39misfile(os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39mjoin(save_dir, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mconfig.json\u001b[39m\u001b[38;5;124m'\u001b[39m)):\n\u001b[1;32m--> 106\u001b[0m         obj\u001b[38;5;241m.\u001b[39mload(save_dir, verbose\u001b[38;5;241m=\u001b[39mverbose, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m    107\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m    108\u001b[0m         obj\u001b[38;5;241m.\u001b[39mload(metapath, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\hanlp\\common\\keras_component.py:215\u001b[0m, in \u001b[0;36mKerasComponent.load\u001b[1;34m(self, save_dir, logger, **kwargs)\u001b[0m\n\u001b[0;32m    213\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mload_config(save_dir)\n\u001b[0;32m    214\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mload_vocabs(save_dir)\n\u001b[1;32m--> 215\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbuild(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mmerge_dict(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconfig, training\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m, logger\u001b[38;5;241m=\u001b[39mlogger, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs, overwrite\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, inplace\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m))\n\u001b[0;32m    216\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mload_weights(save_dir, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m    217\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mload_meta(save_dir)\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\hanlp\\common\\keras_component.py:225\u001b[0m, in \u001b[0;36mKerasComponent.build\u001b[1;34m(self, logger, **kwargs)\u001b[0m\n\u001b[0;32m    223\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mbuild\u001b[39m(\u001b[38;5;28mself\u001b[39m, logger, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m    224\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtransform\u001b[38;5;241m.\u001b[39mbuild_config()\n\u001b[1;32m--> 225\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmodel \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbuild_model(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mmerge_dict(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconfig, training\u001b[38;5;241m=\u001b[39mkwargs\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtraining\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m),\n\u001b[0;32m    226\u001b[0m                                                loss\u001b[38;5;241m=\u001b[39mkwargs\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mloss\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)))\n\u001b[0;32m    227\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtransform\u001b[38;5;241m.\u001b[39mlock_vocabs()\n\u001b[0;32m    228\u001b[0m     optimizer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbuild_optimizer(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconfig)\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\hanlp\\components\\taggers\\transformers\\transformer_tagger_tf.py:34\u001b[0m, in \u001b[0;36mTransformerTaggerTF.build_model\u001b[1;34m(self, transformer, max_seq_length, **kwargs)\u001b[0m\n\u001b[0;32m     33\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mbuild_model\u001b[39m(\u001b[38;5;28mself\u001b[39m, transformer, max_seq_length, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m tf\u001b[38;5;241m.\u001b[39mkeras\u001b[38;5;241m.\u001b[39mModel:\n\u001b[1;32m---> 34\u001b[0m     model, tokenizer \u001b[38;5;241m=\u001b[39m \u001b[43mbuild_transformer\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtransformer\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_seq_length\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtransform\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtag_vocab\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtagging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[0;32m     35\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtransform\u001b[38;5;241m.\u001b[39mtokenizer \u001b[38;5;241m=\u001b[39m tokenizer\n\u001b[0;32m     36\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m model\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\hanlp\\layers\\transformers\\loader_tf.py:11\u001b[0m, in \u001b[0;36mbuild_transformer\u001b[1;34m(transformer, max_seq_length, num_labels, tagging, tokenizer_only)\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mbuild_transformer\u001b[39m(transformer, max_seq_length, num_labels, tagging\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, tokenizer_only\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m):\n\u001b[1;32m---> 11\u001b[0m     tokenizer \u001b[38;5;241m=\u001b[39m \u001b[43mAutoTokenizer_\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfrom_pretrained\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtransformer\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     12\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m tokenizer_only:\n\u001b[0;32m     13\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m tokenizer\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\hanlp\\layers\\transformers\\pt_imports.py:65\u001b[0m, in \u001b[0;36mAutoTokenizer_.from_pretrained\u001b[1;34m(cls, pretrained_model_name_or_path, use_fast, do_basic_tokenize)\u001b[0m\n\u001b[0;32m     63\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m use_fast \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m do_basic_tokenize:\n\u001b[0;32m     64\u001b[0m     warnings\u001b[38;5;241m.\u001b[39mwarn(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m`do_basic_tokenize=False` might not work when `use_fast=True`\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m---> 65\u001b[0m tokenizer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mcls\u001b[39m\u001b[38;5;241m.\u001b[39mfrom_pretrained(get_tokenizer_mirror(transformer), use_fast\u001b[38;5;241m=\u001b[39muse_fast,\n\u001b[0;32m     66\u001b[0m                                 do_basic_tokenize\u001b[38;5;241m=\u001b[39mdo_basic_tokenize,\n\u001b[0;32m     67\u001b[0m                                 \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39madditional_config)\n\u001b[0;32m     68\u001b[0m tokenizer\u001b[38;5;241m.\u001b[39mname_or_path \u001b[38;5;241m=\u001b[39m transformer\n\u001b[0;32m     69\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m tokenizer\n",
      "File \u001b[1;32mc:\\Users\\COLORFUL\\python_virtualenv\\hanlp_py39\\lib\\site-packages\\transformers\\tokenization_utils_base.py:2046\u001b[0m, in \u001b[0;36mPreTrainedTokenizerBase.from_pretrained\u001b[1;34m(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, trust_remote_code, *init_inputs, **kwargs)\u001b[0m\n\u001b[0;32m   2043\u001b[0m \u001b[38;5;66;03m# If one passes a GGUF file path to `gguf_file` there is no need for this check as the tokenizer will be\u001b[39;00m\n\u001b[0;32m   2044\u001b[0m \u001b[38;5;66;03m# loaded directly from the GGUF file.\u001b[39;00m\n\u001b[0;32m   2045\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mall\u001b[39m(full_file_name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01mfor\u001b[39;00m full_file_name \u001b[38;5;129;01min\u001b[39;00m resolved_vocab_files\u001b[38;5;241m.\u001b[39mvalues()) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m gguf_file:\n\u001b[1;32m-> 2046\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mEnvironmentError\u001b[39;00m(\n\u001b[0;32m   2047\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCan\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt load tokenizer for \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mpretrained_model_name_or_path\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m. If you were trying to load it from \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   2048\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mhttps://huggingface.co/models\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m, make sure you don\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt have a local directory with the same name. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   2049\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mOtherwise, make sure \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mpretrained_model_name_or_path\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m is the correct path to a directory \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   2050\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcontaining all relevant files for a \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mcls\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m tokenizer.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   2051\u001b[0m     )\n\u001b[0;32m   2053\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m file_id, file_path \u001b[38;5;129;01min\u001b[39;00m vocab_files\u001b[38;5;241m.\u001b[39mitems():\n\u001b[0;32m   2054\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m file_id \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m resolved_vocab_files:\n",
      "\u001b[1;31mOSError\u001b[0m: Can't load tokenizer for 'uer/albert-base-chinese-cluecorpussmall'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'uer/albert-base-chinese-cluecorpussmall' is the correct path to a directory containing all relevant files for a BertTokenizer tokenizer.\n=================================ERROR LOG ENDS================================="
     ]
    }
   ],
   "source": [
    "# ner = hanlp.load(hanlp.pretrained.ner.MSRA_NER_ELECTRA_SMALL_ZH) # MSRA_NER_ELECTRA_SMALL_ZH\n",
    "# ner = hanlp.load(hanlp.pretrained.ner.MSRA_NER_BERT_BASE_ZH)\n",
    "ner = hanlp.load(hanlp.pretrained.ner.MSRA_NER_ALBERT_BASE_ZH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "elA_UyssOut_"
   },
   "source": [
    "## 命名实体识别"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wxctCigrTKu-"
   },
   "source": [
    "命名实体识别任务的输入为已分词的句子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('周杰伦', 'NR', 0, 3), ('北京', 'NS', 11, 13), ('天安门', 'NS', 13, 16)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner(list('周杰伦2025年12月北京天安门走起来'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Zo08uquCTFSk",
    "outputId": "864da076-7113-4685-e27a-1856e69bdd2a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[('2021年', 'DATE', 0, 1)], [('北京', 'ORGANIZATION', 2, 3), ('立方庭', 'LOCATION', 3, 4), ('自然语义科技公司', 'ORGANIZATION', 5, 9)]]\n"
     ]
    }
   ],
   "source": [
    "print(ner([[\"2021年\", \"HanLPv2.1\", \"为\", \"生产\", \"环境\", \"带来\", \"次\", \"世代\", \"最\", \"先进\", \"的\", \"多\", \"语种\", \"NLP\", \"技术\", \"。\"], [\"阿婆主\", \"来到\", \"北京\", \"立方庭\", \"参观\", \"自然\", \"语义\", \"科技\", \"公司\", \"。\"]], tasks='ner*'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每个四元组表示`[命名实体, 类型标签, 起始下标, 终止下标]`，下标指的是命名实体在单词数组中的下标。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义词典"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "自定义词典是NER任务的成员变量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print(ner.dict_whitelist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 白名单词典\n",
    "白名单词典中的词语会尽量被输出。当然，HanLP以统计为主，词典的优先级很低。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('2021年', 'DATE', 0, 1),\n",
       " ('138', 'INTEGER', 4, 5),\n",
       " ('午饭后', 'TIME', 8, 10),\n",
       " ('2点45', 'TIME', 10, 11),\n",
       " ('44', 'INTEGER', 14, 15)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner.dict_whitelist = {'午饭后': 'TIME'}\n",
    "ner(['2021年', '测试', '高血压', '是', '138', '，', '时间', '是', '午饭', '后', '2点45', '，', '低血压', '是', '44'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 强制词典\n",
    "如果你读过[《自然语言处理入门》](http://nlp.hankcs.com/book.php)，你就会理解BMESO标注集，于是你可以直接干预统计模型预测的标签，拿到最高优先级的权限。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('浙江', 'LOCATION', 2, 3), ('金华', 'LOCATION', 3, 4), ('金华', 'PERSON', 10, 11)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner.dict_tags = {('名字', '叫', '金华'): ('O', 'O', 'S-PERSON')}\n",
    "ner(['他', '在', '浙江', '金华', '出生', '，', '他', '的', '名字', '叫', '金华', '。'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 黑名单词典\n",
    "黑名单中的词语绝对不会被当做命名实体。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('浙江', 'LOCATION', 2, 3)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner.dict_blacklist = {'金华'}\n",
    "ner(['他', '在', '浙江', '金华', '出生', '，', '他', '的', '名字', '叫', '金华', '。'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "ner_stl.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "hanlp_py39",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
